Display techniques for three-dimensional virtual reality

ABSTRACT

A limitation of using two-dimensional images, such as videos or photographs, to represent portions of a three-dimensional world occurs when the user moves within the world and views the world from a location different than from the original context of the two-dimensional image, i.e., from a location different than the image&#39;s ideal viewing point (IVP). View changes result in the image not aligning well with the surrounding objects of the three-dimensional world. This limitation is overcome by distorting the two-dimensional image so as to adjust the image&#39;s vanishing point(s) in accordance with the movement of the user using a pyramidic panel structure. In this manner, as the user moves away from the ideal viewing point, the distortions act to limit the discontinuities between the two-dimensional image and its surroundings. Various pyramidic panel structures may be constructed, including an octahedral panel structure which surrounds the user&#39;s view point so as to provide a so-called “plenoptic” view of the world. Also, to minimize the depth profile of the pyramidic panel structure, the structure may be segmented into sections and each section translated towards, or away from, the user&#39;s viewpoint. Also, a hierarchical image resolution may be used, with portions of the image near the center or vanishing point having a higher resolution than portions of the image near its perimeter.

CROSS-REFERENCE TO RELATED APPLICATION

This application is related to co-pending U.S. patent application Ser.No. 09/191,012, filed concurrently herewith (Case Edmark-6). Also, thisapplication is a continuation-in-part of U.S. patent application Ser.No. 09/160,758 filed Sep. 25, 1998 now U.S. Pat. No. 6,236,402 (CaseEdmark-5), which is a continuation-in-part of U.S. patent applicationSer. No. 09/107,059 filed Jun. 30, 1998 now U.S. Pat. No. 6,229,548(Case Edmark-2). The above-identified co-pending applications, which arecommonly assigned, are incorporated herein by reference.

TECHNICAL FIELD

This invention relates to the integration of three-dimensional computergraphics and a two-dimensional image to provide a realisticthree-dimensional virtual reality experience.

BACKGROUND OF THE INVENTION

The display of a three-dimensional virtual reality world to a userrequires considerable computation power, and it is typically costly todevelop the necessary highly detailed models required for doing so. Inorder to simplify the problem, two-dimensional images, such as videos orphotographs, may be used to represent or simulate portions of thethree-dimensional world. A great reduction in computation power and costcan be achieved by such an arrangement.

SUMMARY OF THE INVENTION

A limitation of such a world occurs when a user moves within the worldand views the world from a location different than the original contextof a two-dimensional image which has been carefully calibrated to “fitinto” the world. View changes, such as from a location different thanthe image's ideal viewing point, result in the image not aligning orfitting well with the surrounding objects of the three-dimensionalworld. I have recognized that, in accordance with the principles of theinvention, viewpoint changes may be dealt with by distorting thetwo-dimensional image so as to adjust the image's vanishing point(s) inaccordance with the movement of the user using a novel “pyramidic panelstructure.” In this manner, as the user moves away from the idealviewing point, the distortions act to limit the discontinuities betweenthe two-dimensional image and the surroundings of the world. Variouspyramidic panel structures may be constructed, including an octahedralpanel structure which surrounds the user's view point so as to providethe user with a so-called “plenoptic” view of the world.

In another aspect of the present invention, the pyramidic panelstructure may be segmented into sections, each translated towards oraway from the user's viewpoint and then scaled, so as to minimize thedepth profile of the pyramidic panel structure. In yet still anotheraspect of the present invention, a hierarchical image resolution may beused, with portions of the image near the center of the image having ahigher resolution than portions of the image near its perimeter.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 shows an example of that which a user sees when a user views theworld from the ideal viewing point for a two-dimensional imagerepresenting a portion of the world;

FIG. 2 shows an example of that which a user sees when a user moveswithin the world of FIG. 1 and views the two-dimensional image from alocation different than the image's ideal viewing point, without the useof the present invention;

FIG. 3 shows an exemplary process, in accordance with the principles ofthe invention, for distorting the two-dimensional image using apyramidic panel structure so as to adjust the image's vanishing point inaccordance with the movement of the user;

FIGS. 4 and 5 depict the pyramidic panel structure of the presentinvention for distorting the two-dimensional image so as to adjust theimage's vanishing point, in accordance with the movement of the user;

FIGS. 6A-B depict examples of that which a user sees when a user viewsthe world from a location left of the image's ideal viewing point,without and with the use of the present invention, respectively;

FIGS. 7A-B depict examples of that which a user sees when a user viewsthe world from a location above the image's ideal viewing point, withoutand with the use of the present invention, respectively;

FIGS. 8A-B depict examples of that which a user sees when a user viewsthe world from a location toward the front and the right of the image'sideal viewing point, without and with the use of the present invention,respectively;

FIG. 9 depicts another embodiment of the pyramidic panel structure ofthe present invention;

FIGS. 10-12 depict the pyramidic panel structure of FIG. 9 for variouslocations of the user's current viewing point, V;

FIG. 13 shows an exemplary process, in accordance with the principles ofthe invention, for distorting a two-dimensional image using anarticulated pyramidic panel structure so as to adjust multiple vanishingpoints in the image, in accordance with the movement of the user;

FIG. 14 depicts an example of the articulated pyramidic panel structureof the present invention;

FIG. 15 depicts an example of that which a user sees when a user viewsthe world from a location away from the ideal viewing point of thetwo-dimensional image, with the use of the articulated pyramidic panelstructure of the present invention;

FIGS. 16 and 17 depict side and front views, respectively, of thepyramidic panel structure of FIG. 5 with each panel segmented into aplurality of sections;

FIG. 18 depicts the pyramidic panel structure of FIG. 5, with each panelsegmented into a plurality of sections having its centers located on thesurface of a predetermined plane;

FIG. 19 depicts the pyramidic panel structure of FIG. 5, with each panelsegmented into a plurality of sections and each section translated adifferent distance toward the user's view point, V;

FIG. 20 depicts the screen of the pyramidic panel structure of thepresent invention segmented into four triangular sections onto which thecorresponding portions of the two-dimensional image is displayed;

FIG. 21 depicts the screen of FIG. 20 with the corresponding portions ofthe image textured-mapped onto the pyramidic panels in accordance withthe present invention; and

FIG. 22 depicts the screen of FIG. 20 with the two-dimensional imagesegmented into a plurality of sections in accordance with another aspectof the present invention.

DETAILED DESCRIPTION

To better understand the invention, FIGS. 1-2 show examples of thatwhich a user sees when the user moves within a three-dimensional virtualreality world (x,y,z) and views a two-dimensional image (x,y)representing a portion of the world from a location at the image's idealviewing point (IVP), and then from a different location, i.e., alocation different than the original context of the image. It should beunderstood that the two-dimensional image has been carefully calibratedto “fit into” the surroundings of the world. For simplification ofterminology purposes, we shall use the term two-dimensional image todenote either a video clip or a photograph. In accordance with theprinciples of the invention, as the user moves away from the idealviewing point, discontinuities between the two-dimensional image and itssurroundings are minimized by distorting the image according to themovement of the user.

FIG. 1 shows an exemplary three-dimensional reality world 105, which isa bicycle path in a park, e.g., Central Park in New York City. Inrepresenting world 105, the present invention exploits a characteristiccommon for images consisting of views looking down the center of roads,streets or paths, which is that they may be treated as perspective,corridor-like images, with features closer to the center of the imagebeing farther away from the viewer along the z-axis. Accordingly, thebicycle path or road and its immediate vicinity are treated as a kind ofthree-dimensional, corridor-like image whose floor is formed by theroadbed, whose ceiling is formed by the sky, and whose sidewalls areformed by the roadside objects. In this manner, the principles of asimple point perspective can be used for distorting the landscape imagein accordance with the movement of the viewer, as discussed hereinbelow.

World 105 is divided into two portions, screen or panel 110 on which isshown or displayed a two-dimensional image 115, such as a stillphotograph, picture, or a current frame of a video clip; and theremainder of the world 120, which is represented using computer graphicstechniques, and is thus referred to herein as computer graphics (CGPart) 120. Within CG Part 120 there are various synthetic,three-dimensional landscapes or objects modeled in, for example, theVirtual Reality Modeling Language (VRML). Two-dimensional image 115simulates landscape or terrain portions of the world 105, here a virtualroad or course 125 for walking, running or pedaling a bicycle.

Note that although three-dimensional world 105 cannot be actuallyrendered in a two-dimensional plane (x,y), it can be projected to anddisplayed on a two-dimensional plane so as to appear to have threedimensions (x,y,z). Accordingly, the techniques of the present inventionare preferably employed with computers and software, which aresufficiently sophisticated to display images on a two-dimensional planeas having three dimensions. Note also that to make the world lookrealistic, computer graphics display techniques use the z component ofobjects to scale accordingly the x and y components as a function of itsdistance (z-axis) to the user's viewpoint.

Two-dimensional image 115 is carefully placed, cropped and sized toachieve continuity with the surrounding environment of the CG Part 120.Note that the image is clipped so that the left and right edges of theroad in CG Part 120 pass through the left and right bottom corners ofthe road, respectively, in image 115. This clipping ensures that theroadbed maps to the floor of the hypothetical corridor. In so doing,portions at the boundary between two-dimensional image 115 and CG part120 are co-planar, i.e., at the same distance away along the z-axis fromthe user's viewpoint. In “fitting” two-dimensional image 115 to CG part120, however, there exits only one viewpoint from which that image'scontent properly corresponds to the surrounding environment of CG Part120. This unique location is called the image's ideal viewing point(IVP). In FIG. 1, two-dimensional image 115 is seen from its idealviewing point, and from this view, image 115 aligns well with thesurrounding objects of CG Part 120.

Users, however, rarely view image 115 only from its idea viewing point.As the user moves within world 105, such as left or right of road 125,as they round curves, or move closer to or farther from the image, theysee image 115 from positions other than its ideal viewing point. Absentthe use of the present invention, such viewpoint changes would causeobjects or features within image 115 to align improperly with thesurrounding environment, as further illustrated in FIG. 2.

In accordance with the principles of the invention, however, screen orpanel 110 uses a display structure called a “pyramidic panel structure”for displaying two-dimensional image 115 within the surroundingthree-dimensional space of the CG Part 105 so as to deal with viewpointchanges. The transformations associated with the pyramidic panelstructure dynamically distort two-dimensional image 115 according toviewer's position so as to adjust the image's vanishing point with theviewer's movement. As the viewer moves from the image's ideal viewingpoint, these distortions act to limit discontinuities between image 115and the surroundings of CG Part 120.

FIG. 3 shows an exemplary process in accordance with the principles ofthe invention for distorting two-dimensional image 115 so as to adjustits vanishing point in accordance with the viewer's position. Theprocess is entered at step 130 whenever it is determined that theviewer's position has changed.

Using the virtual world's road model of the CG Part 105, a vector,{overscore (C)}, corresponding to the direction of road 125 is projectedat step 135 from the image's ideal viewing point, IVP, to panel orscreen 110 on which is displayed image 115. Note that the panel istwo-dimensional, but represents three-dimensional space with objectsnearer the center of the image being farther away from the plane of theviewer. The panel structure is shown in FIG. 4. The point ofintersection with screen or panel 110 is the image's vanishing point, P.Note, however, that the vanishing point may be set visually by the user,if desired, or by other suitable computer graphics processing techniquesknown in the art. Next, in step 140, screen or panel 110 is segmentedinto four triangular regions 145 ₁₋₄, one for each of the regionsbordering CG Part 120, with the intersection point of the four regionslocated at the vanishing point, P.

Thereafter in step 150, the current viewpoint of the user, V, isdetermined, and a vector {overscore (T)} projected from the idealviewing point, IVP, to the viewer's current location, V. In accordancewith the principles of the invention, as the viewer moves, a newvanishing point P′ is calculated as P′=P+{overscore (T)}. The fourtriangular regions 145 ₁₋₄ are distorted in the three-dimensional spaceof the virtual world at step 155 to represent the mapping of objectsnearer the center of the image being displaced farther away from theviewpoint of the user. The four triangular regions intersect at the newvanishing point P′ and form so-called “pyramidic panels” 145′₁₋₄. Thisis illustrated in FIG. 5. At step 160, the corresponding imagesdisplayed in regions 145 ₁₋₄ are then “texture-mapped” onto pyramidicpanels 145′₁₋₄, respectively. In this manner, as the viewer moves awayfrom the image's ideal viewing point, IVP, distortions in the imageresulting from moving the image's vanishing point from P to P′ act tolimit the discontinuities between image 115 and the surroundings withinCG Part 105.

In the exemplary illustration of FIG. 5, distorting image 115 so as tomove the vanishing point from P to P′ results in pyramidic panelstructure forming a four-sided pyramid. Note that its base is fixed andcorresponds to original screen or panel 110, with its peak located atP′, which moves in concert with the viewer's current location, V. As theuser's viewpoint moves closer to and farther from the image, the image'svanishing point accordingly moves farther from and closer to the user'sviewpoint, respectively.

FIGS. 6 through 8 compare the display of two-dimensional image 115 onscreen or panel 110 with the display of the same image using the“pyramidic” panels of the present invention. More specifically, FIGS.6A, 7A and 8A depict viewing two-dimensional image 115 at a locationfrom the left, above, and in front and to the right of the image's idealviewing point, IVP, respectively, without the use of the presentinvention. In these latter figures, note that there are discontinuitiesbetween the edges of the road and the three-dimensional space of CG Part105. FIGS. 6B, 7B and 8C depict the same two-dimensional image distortedand texture-mapped onto pyramidic panels 145′₁₋₄, in accordance with theprinciples of the invention. Note that in these latter figures, thediscontinuities in the road edge have been substantially eliminated.

Referring now to FIG. 9, there is illustrated another embodiment of thepresent invention which is similar to that of FIG. 5, except that twoco-located image panels 110, and 110′ are employed. Image panels 110 and110′ are of the same or similar size, shape and orientation, but face inopposite directions. As discussed herein below, image panels 110, 110′are employed to simulate a so-called “plenoptic” view, i.e., everythingthat is viewable in virtual world 120 to the user from his currentviewpoint, V. More specifically, displayed on image panel 110 istwo-dimensional image 115, which simulates portions of the variousthree-dimensional landscapes or objects that can be observed from theuser's current viewpoint, V, in the virtual space defined by therectangular coordinates (±x, ±y, −z). However, displayed on image panel110′ is two-dimensional image 115′ which simulates the remaining portionof the “plenoptic view” or those three-dimensional landscapes or objectswhich can be observed from the user's current viewpoint, V, in thevirtual space defined by the rectangular coordinates (±x, ±y, z). Assuch, what is observed by the user in virtual world 105, as a result ofmapping image 115 or 115′ onto image panel 110 or 110′, respectively, isdependent on the direction from which the user views the virtual world.Using two image panels in effect entirely surrounds the user'sviewpoint, V, within the virtual world so as to provide a “plenopticview.” Not only is the user able to pan along a 360° field of view alongthe x,y and z axes, but advantageously is also able to move within andaround the world without noticeably observing any discontinuities in theworld.

Importantly, panels 110 and 110′ are similarly segmented into triangularregions 145 ₁₋₄ and 145 _(1′-4′), respectively, and use the “pyramidicpanel structure” discussed herein above for displaying thetwo-dimensional images so as to deal with viewpoint changes. Recall thatthe transformations associated with the pyramidic panel structuredynamically distort the two-dimensional image according to the viewer'sposition so as to adjust the image's vanishing point in accordance withthe viewer's movement. As the viewer moves from the image's idealviewing point, these distortions act to limit discontinuities betweenthe image, and the surroundings of the virtual world.

Note, however, that the respective views of two-dimensional images 115,115′ should be carefully selected so that the edges of the two viewscorrespond to the same location in virtual world 105. In this manner,the two image panels can be attached at their edges, but facing inopposite directions. As discussed above, each panel is distorted in thethree-dimensional space of the virtual world so as to represent themapping of objects nearer the center of the image being displacedfarther away from the viewpoint of the user. As the user moves from theideal viewing point, IVP, of image 115, regions 145 ₁₋₄ distort intopanels 145′₁₋₄, forming a four-sided pyramid. And, likewise, moving fromthe ideal viewing point, IVP′ of image 115′ results in regions 145_(1′-4′), distorting into panels 145″₁₋₄, and forming another four-sidedpyramid.

Accordingly, when the two image panels distorts in response to theuser's viewpoint, a textured-mapped, octahedral panel structure isformed surrounding the user's view point, V. This latter aspect of thepresent invention may be more fully appreciated by now referring toFIGS. 10-12. As the user's viewpoint, V, is moved along the z-axis awayfrom two-dimensional image 115, the peaks (P′,P″) of each pyramidicstructure moves accordingly along the same direction, as shown in FIG.10. Now referring to FIG. 11, when the user's viewpoint for image 115,however, is located at the image's ideal viewing point, IVP, panels145′₁₋₄ then collapse onto panel 110. Similarly, this occurs for panels145″₁₋₄ when the user's viewpoint is located at the ideal viewing pointIVP′, for image 115′, as further illustrated in FIG. 12.

In another embodiment of the present invention, a modified pyramidicpanel structure may be used to deal with two-dimensional imagescontaining curved roads, streets, paths and other corridor-like imagescontaining multiple rather than a single vanishing point. In this lattercase, screen or panel 110 is segmented using multiple vanishing pointsto form a so called “articulated pyramidic panel structure.” Thetransformations associated with the articulated pyramidic panelstructure dynamically distort different portions of two-dimensionalimage 115 according to viewer positions so as to adjust the differentvanishing points of the image with the viewer's movement. Likewise, asthe viewer moves from the image's ideal viewing point, these distortionsact to limit the discontinuities between two-dimensional image 115 andthe surroundings of CG Part 120.

FIG. 13 shows an exemplary process in accordance with the principles ofthe invention for distorting two-dimensional image 115 using anarticulated pyramidic panel structure. Again, the process is entered atstep 170 whenever it is determined that the viewer's position haschanged. In general, curve road 125 is treated as two straight corridorsplaced end-to-end, extending back from screen or panel 110. Eachcorridor represents a different portion of road 125 in thethree-dimensional space of world 105, with features nearer the center ofthe image being farther away from the user's viewpoint.

Using the virtual world's road model of the CG Part 105, correspondingdirectional vectors C₁ and C₂ of the corridors are determined at step175. Note that portion of the road nearer to the user's viewpoint isrepresented by C₁, and the portion farther away is represented by C₂.Next, in step 180, using the vectors C₁ and C₂, the correspondingvanishing points P₁ and P₂ are determined, respectively, for eachcorridor by projecting those vectors from the image's ideal viewingpoint, IVP. Alternatively, vanishing points P₁ and P₂ may be determinedvisually by the user, or by some other suitable means known in the art.In step 185, using the first corridor's vanishing point, P₁, a first setof pyramidic panels 190 ₁₋₄ are constructed to intersect at vanishingpoint, P₁, as shown in FIG. 14.

Now at step 195, a coupling ratio α is calculated according to thefollowing equation: α=l/(l+d), where 1 is the length of the firstcorridor, and d is the distance between the image's ideal view point(IVP) and the base of pyramidic panels 190 ₁₋₄. Each line segmentconnecting a corner of the base to vanishing point P₁, is then dividedinto two segments by a point placed according to the coupling ratio, α.More specifically, the length l′ of each line segment from the corner ofthe base of panels 190 ₁₋₄ to this point is given by l′=αl″, where l″ isthe total length of the segment between the corner of the panel and thevanishing point, P₁. These four points labeled Q1 through Q4 areconnected to form the base of a second set of smaller pyramidic panels200 ₁₋₄ embedded within the larger panels (step 205), as furtherillustrated in FIG. 14. The intersection point of pyramidic panels 200₁₋₄ is then moved from P₁ to vanishing point, P₂.

For the articulated pyramidic panel structure, the current viewpoint ofthe user, V, is determined, and a vector {overscore (T)} projected fromthe ideal viewing point, IVP, to the viewer's current location, V (step210). As the viewer moves, a new vanishing point P′₂ is calculated asP′₂=P₂+{overscore (T)} at step 215, and panels 200 ₁₋₄ are thendistorted so as to intersect at P′₂. As the viewer move, the fourinternal points Q1 through Q4 are mapped with the viewer's movement toQ1′ through Q4′, respectively, in accordance with the followingrelationship: Q′_(i)=Q_(i)+α{overscore (T)}, at step 220. Note thatdoing so, accordingly distorts the first set of pyramidic panels 190₁₋₄. At step 225, the corresponding images in original panels are thentexture-mapped into articulated pyramidic panels 190 ₁₋₄ and 200 ₁₋₄,which have been distorted in accordance with the movement of the viewer.Note that to unambiguously texture-map onto panels 190 ₁₋₄, these panelsare each subdivided into two triangular subregions and thentexture-mapped. Shown in FIG. 15 is image 115 seen from a location awayfrom the image's ideal viewing point, using the articulated pyramidicpanel structure of the present invention.

Note that the above articulated pyramidic panel structure may also usemore than two sets of pyramidic panel structures. Instead of treatingthe curve road as two straight corridors, multiple corridors may beemployed, each placed end-to-end and extending back from screen or panel110. Likewise, each corridor represents a different portion of road 125in the three-dimensional space of world 105, with features nearer thecenter of the image being farther away from the user's viewpoint. Insuch a case, each set of articulated pyramidic panels are formedreitererately using the above described procedure.

Referring to FIGS. 16 and 17, there is shown a third embodiment of thepresent invention which is similar to that of FIG. 5 and in which“pyramidic panels” 145′₁, 145′₂, 145′₃ and 145′₄ have been nowmulti-segmented into sections 205 ₁₋₄, 210 ₁₋₄, 205′₁₋₄, and 210′₁₋₄,respectively, with the images in original panels 145 ₁₋₄ thentexture-mapped into the corresponding translated sections of thepyramidic panel structure, as discussed herein below. It should berecalled that the pyramidic panel structure represents thethree-dimensional mapping (x,y,z) of two-dimensional image 115 ontoimage screen or panel 110 (x,y). Advantageously, the embodiment of FIGS.16-17 minimizes the depth profile of the pyramidic panel structure alongthe z-axis. Unlike the embodiment of FIG. 5, the depth profile of thisthird embodiment does not substantially vary with changes in the user'sviewpoint, V. In the exemplary embodiment of FIG. 5, recall thatdistorting image 115 so as to move the vanishing point from P to P′results in the pyramidic panel structure forming a four-sided pyramid.The base of the pyramid is fixed and corresponds to original screen orpanel 110, with its peak located at P′ and moves in concert with theviewer's current location, V. As the user's viewpoint moves along thez-axis closer to and farther from two-dimensional image 115, the image'snew vanishing point P′ moves farther from and closer to the user'sviewpoint, respectively. This latter movement causes the depth profilealong the z-axis of the pyramidic panel structure to vary accordingly.Unfortunately, this variation in depth profile can undesirably and/orunexpectedly occlude from the user's view objects in the virtual world,or cause objects to occlude other features in the virtual world inasmuchas the corresponding images in the panels are distorted, as discussedabove herein.

To obviate the aforementioned problem, “pyramidic panels” 145′₁, 145′₂,145′₃ and 145′₄ have been multi-segmented into sections 205 ₁₋₄, 210₁₋₄, 205′₁₋₄, and 210′₁₋₄ respectively. Each section is then translatedalong the z-axis to a predetermined distance towards or away from theuser's viewpoint, V, but importantly of the same orientation as theoriginal section. For example, segmented sections 205 ₁₋₄ and 205′₁₋₄may each have one of its outer edge along the x-axis translated to lieon the x,y plane of screen or panel 110, as shown in phantom in FIG. 16.As the user moves to a new viewpoint, each section in effect pivotsabout that edge along the x-axis, which edge lies on the surface ofpanel 110. Similarly, section 210 ₁₋₄ and 210′₁₋₄ may each have one ofit outer edge along the y-axis lying on the surface of panel 110.Alternatively, sections 205 ₁₋₄ and 205′₁₋₄ may be centered along panel110, as depicted in FIG. 18. Likewise, sections 210 ₁₋₄ and 210′₁₋₄ maybe similarly translated, but for the sake of clarity are not shown inFIGS. 16 and 18.

Still further, each of sections 205 ₁₋₄ and 205′₁₋₄ may in effect berotated or pivoted about its other edge along the x-axis as the usermoves to a new viewpoint, V, or, in general, about an axis parallel withan edge along the x-axis of the corresponding section. Again, thislatter axis may, but does not have to, lie on the surface of panel 110.Regardless of the segmenting method chosen, however, translating eachsection towards or away from the user's viewpoint significantly reducesthe depth profile of the pyramidic panel structure along the z-axis,such as depicted in FIG. 18 from, for example, T_(2 to T) ₁.

In still another embodiment of the present invention, sections 205 ₁₋₄and 205′₁₋₄ may each be translated a different distance along thez-axis, as illustrated in FIG. 19. Although not shown, sections 210 ₁₋₄and 210′₁₋₄ may likewise be translated. Those skilled in the art willreadily understand that doing so advantageously allows the user'sviewpoint, V, to extend in front of panel 110 inasmuch as segmentedsections corresponding to the image's center may be offset and locatedcloser to the user's viewpoint, V, than the outer sections.

Also, note that segmenting the pyramidic panel structure into a greaternumber of smaller sections accordingly only further reduces the depthprofile, which asymptotically approaches a zero thickness. It iscontemplated that the number of sections that the panel structure isdivided into may be chosen empirically based on image content as well asthe user's range of movement within the virtual world. Preferably,however, the panel structure is dynamically segmented in a reiterativemanner. For example, once a user has chosen the maximum desired depthfor the panel structure along the z-axis to minimize occlusion, eachpanel is then reiteratively segmented into a greater number of smallersections until the depth profile is reduced to the maximum depth profiledesired.

In accordance with the principles of the invention, it should be clearlyunderstood, however, that to maintain the apparent integrity oftwo-dimensional image 115 when texture-mapping the image onto thesegmented sections, each segmented sections 205 ₁₋₄, 205′₁₋₄, 210 ₁₋₄and 210′₁₋₄ is scaled accordingly with respect to the user's currentviewpoint, V, so as to appear to be of the same size as the originalcorresponding section. This scaling or transform is given by:$S_{t} = {S_{p}\frac{T_{t}}{T_{p}}}$

where S_(p) is the size of the original pyramidic section; S_(t) is thesize of the translated, segmented pyramidic panel section; T_(p) isdistance to the original pyramidic section from the user's viewpoint, V;and T_(t) is the distance to the translated, segmented pyramidicsection. In other words, each segmented, translated section is scaled bythe ratio $\frac{T_{t}}{T_{p}}.$

Of course, as the user moves within the world, pyramidic panels 145 ₁₋₄are accordingly re-segmented, translated, and then scaled with respectto the user's new viewpoint, V. Then, the images in original panels 145₁₋₄ are again accordingly texture-mapped into the correspondingtranslated sections 205 ₁₋₄, 205′₁₋₄, 210 ₁₋₄ and 210′₁₋₄ of thepyramidic panel structure.

In the above embodiments, distorting two-dimensional image 115 accordingto the movement of the user's viewpoint results in different portions ofthe image being accordingly “expanded” or “compressed” whentexture-mapping image 115 onto the corresponding pyramidic panels, asdiscussed above herein. Also, as the user's viewpoint moves along thez-axis closer to two-dimensional image 115, the image is accordinglyscaled or “enlarged” to make objects in the image appear closer to theuser. Note that doing so requires scaling the x and y components of theobjects accordingly as a function of their distance (z-axis) to theuser's viewpoint. To illustrate these later aspects of the presentinvention, shown in FIG. 20 is screen 110 with two-dimensional image 115displayed therein having a resolution of M×N pixels, e.g., 1200×1200.Now, as shown in FIG. 21, moving along the z-axis closer to the centerof the image and then texture-mapping the corresponding portions ofimage 115 onto pyramidic panels 145′₁₋₄ requires enlarging anddistorting the image such that only a small center portion of theenlarged and distorted image is displayed on screen 110, which in effectlowers the observable image resolution. This is so since the image onlyhas a finite number of pixels, and fewer pixels are displayed on thesame size screen. One solution to this latter problem is using highresolution images which allows the user to travel towards the center ofthe image without losing image quality, but it results in the perimeterof the image having a resolution higher that needed since enlarging theimage causes the perimeter of the image to be outside the field of viewof the observer and, therefore not displayed on the screen. This higherthan needed resolution requires more memory to store image 115 as wellas additional computational time to perform the texture mapping, amongother things. Of course, using lower resolution images would requireless memory and computational time, but it typically leads to poor imagequality near the center of image where the image is typically enlargedor distorted the greatest.

Referring to FIG. 22, there is shown still another embodiment of thepresent invention which obviates the aforementioned problem by using atwo-dimensional image having a “hierarchical image resolution,” therebyallowing segmented portions of the image to have a resolution accordingto its location within the image. In this manner, a fine resolution canbe used for portions of image 115 near the center of the image whereas acoarse resolution can be used for the perimeter, thereby minimizing thetotal texture-map size or pixel-map size. It should be clearlyunderstood that this hierarchical image resolution is equally applicableto any one of the above pyramidic panel structures discussed hereinabove. In this aspect of the present invention, two-dimensional image115 has been segmented into resolution regions 215 ₁₋₄ each having arectangular shape, although other shapes are readily adaptable for thepurposes of this invention. The corresponding portions of image 115within resolution regions 215 ₁₋₄ have different resolutions such thatportions of the image closer to the center of the image have a finerresolution than portions farther away. For example, resolution regions215 ₁₋₄ may have image pixel resolutions of m₁×m₁, m₂×m₂, m₃×m₃ andm₄×m₄, respectively, where m₁≧m₂≧m₃≧m₄. In this manner, distorting orenlarging portions of the image near the center of the image or wherethe image's vanishing point is typically located does not substantiallyaffect the observed image resolution inasmuch as a greater number ofpixels are contained in the image.

It is contemplated that the number of resolution regions that image 115is segmented into as well as the pixel resolution therein may be, forexample, chosen empirically based on image content as well as the user'srange of movement within the virtual world. For example, portions of theimage near the center can have a resolution of 4800×4800, while portionsnear the perimeter only a resolution of 1200×1200. Furthermore, theresolution for resolution regions 215 ₁₋₄ may be dynamically chosenaccording to the amount of distortion or enlargement performed on theimage. That is, each region is allocated a finer resolution with agreater distortion or enlargement, and vice-a-versa.

Of course, various computer storage techniques may be used to set theimage resolution within regions 215 ₁₋₄. For example, image 115,typically a still photograph, picture or video frame, may be capturedwith a resolution of 4800×4800 pixels or greater, and then a subset ofthose pixels used to achieve a desired lower resolution.

The foregoing merely illustrates the principles of the invention. Itwill thus be appreciated that those skilled in the art will be able todevise various arrangement which, although not explicitly describe orshown herein, embody the principles of the invention and are includedwithin its spirit and scope.

What is claimed is:
 1. A method for use in processing a view of athree-dimensional world in which said world is represented by a firsttwo-dimensional image for viewing objects located in a first portion ofa plenoptic view of said world, and represented by a secondtwo-dimensional image for viewing a second portion of the plenoptic viewof said world, said second portion being the remaining portion of theplenoptic view of said world, comprising the steps of: determining thecurrent user's viewpoint within the three-dimensional world, each ofsaid first and second two-dimensional images being such that featurescloser to a predetermined point of the corresponding image are fartheraway from the user's viewpoint so as to give a portion thereof avanishing point; distorting the first and second two-dimensional imagesso as to move the vanishing points of the portions of the correspondingtwo-dimensional images according to the current user's viewpoint, V; andas the user moves within the three-dimensional world, repeating theabove step so as to limit discontinuities between said first and secondtwo-dimensional images, and the computer graphics.
 2. The method asdefined in claim 1 wherein said predetermined point is substantiallynear the center of the corresponding two-dimensional image.
 3. Themethod as defined in claim 1 wherein said predetermined point issubstantially near the vanishing point of the correspondingtwo-dimensional image.
 4. The method as defined in claim 1 wherein eachof said first and second two-dimensional images is a frame of a video.5. The method as defined in claim 1 wherein each of said first andsecond two-dimensional images is a still picture.
 6. The method asdefined in claim 1 further comprising the step of calibrating said firstand second two-dimensional images as a function of the dimensions of thesurroundings within the world.
 7. A method for use in processing a viewof a three-dimensional world, comprising the steps of: representing saidworld by first and second two-dimensional images for viewing objectslocated in first portion and second portions of a plenoptic view of saidworld, respectively, said first and second two-dimensional images eachincluding an object depicted in perspective, said first and secondtwo-dimensional images being such that features of the object closer toa predetermined point of the corresponding image are farther away from auser's viewpoint; when viewing objects located in said first and secondportions of the plenoptic view of said world, (a) determining a vector,{overscore (C)}, corresponding to the direction of the perspectiveobject in the corresponding two-dimensional image; (b) projectingtowards an image panel the vector, {overscore (C)}, from thecorresponding two-dimensional image's ideal viewing point, IVP, theintersection of said vector, {overscore (C)}, with the image panel beingdenoted as the image's vanishing point, P; (c) segmenting the imagepanel into triangular regions intersecting at the corresponding image'svanishing point, P; (d) determining the current viewpoint, V, of theuser and projecting a vector, {overscore (T)}, from the correspondingimage's ideal viewing point, IVP, to the current viewpoint, V; (e)determining a new vanishing point for the corresponding two-dimensionalimage in accordance with the following relationship P′=P+{overscore(T)}; (f) distorting the triangular regions in the space of thethree-dimensional world such that they intersect at the new vanishingpoint, P′; and (g) texture-mapping the corresponding two-dimensionalimage in the triangular regions onto said distorted triangular regions;and as the user moves within the world repeating the above steps so asto limit discontinuities between said two-dimensional images, and thecomputer graphics.
 8. The method as defined in claim 7 wherein saidpredetermined point is substantially near the center of thecorresponding two-dimensional image.
 9. The method as defined in claim 7wherein said predetermined point is substantially near the vanishingpoint of the corresponding two-dimensional image.
 10. The method asdefined in claim 7 wherein each of said first and second two-dimensionalimages is a frame of a video.
 11. The method as defined in claim 7wherein each of said first and second two-dimensional images is a stillpicture.
 12. The method as defined in claim 7 further comprising thestep of calibrating said first and second two-dimensional images as afunction of the dimensions of the surroundings within the world.
 13. Anapparatus for use in processing a view of a three-dimensional world inwhich said world is represented by a first two-dimensional image forviewing objects located in a first portion of a plenoptic view of saidworld, and represented by a second two-dimensional image for viewingobjects located in a second portion of the plenoptic view of said world,said second portion being the remaining portion of the plenoptic view ofsaid world, said apparatus comprising: means for determining the currentuser's viewpoint within the three-dimensional world, each of said firstand second two-dimensional images being such that features closer to apredetermined point of the corresponding image are farther away from theuser's viewpoint so as to give a portion thereof a vanishing point; andwhen viewing objects in the first and second portions of the plenopticview of said world, and the user moves within the three-dimensionalworld, means for repeatingly distorting the first and secondtwo-dimensional images, respectively, so as to move the vanishing pointsof the portions of the corresponding two-dimensional images according tothe current user's viewpoint.
 14. The method as defined in claim 13wherein said predetermined point is substantially near the center of thecorresponding two-dimensional image.
 15. The method as defined in claim13 wherein said predetermined point is substantially near the vanishingpoint of the corresponding two-dimensional image.
 16. The method asdefined in claim 13 wherein each of said first and secondtwo-dimensional images is a frame of a video.
 17. The method as definedin claim 13 wherein each of said first and second two-dimensional imagesis a still picture.
 18. The method as defined in claim 13 furthercomprising the step of calibrating said first and second two-dimensionalimages as a function of the dimensions of the surroundings within theworld.
 19. A method for use in processing a view of a three-dimensionalworld in which said world is represented by a first two-dimensionalimage mapped on a first panel for viewing objects located in a firstportion of a plenoptic view of said world, and represented by a secondtwo-dimensional image mapped on a second panel for viewing objectslocated in a second portion of the plenoptic view of said world, saidsecond portion being the remaining portion of the plenoptic view of saidworld, comprising the steps of: determining the current viewpoint of theuser, V; dividing the first and second panels into triangular regions;distorting the triangular regions of said first and second panels toform pyramidic panels such that a corresponding vanishing point, P, of aportion of the corresponding two-dimensional image moves as a functionof the current viewpoint of the user; texture-mapping said first andsecond two-dimensional images onto the plurality of sections of thecorresponding pyramidic panels; and as the user moves within thethree-dimensional world, repeating the above steps so as to limitdiscontinuities between said first and second two-dimensional images,and the computer graphics.
 20. The method as defined in claim 19 whereinsaid predetermined point is substantially near the center of thecorresponding two-dimensional image.
 21. The method as defined in claim19 wherein said predetermined point is substantially near the vanishingpoint of the corresponding two-dimensional image.
 22. The method asdefined in claim 19 further comprising displaying the correspondingdistorted two-dimensional image merged with the first portion of saidworld that is modeled as computer graphics.
 23. The method as defined inclaim 19 wherein each of said first and second two-dimensional images isa frame of a video.
 24. The method as defined in claim 19 wherein eachof said first and second two-dimensional images is a still picture. 25.The method as defined in claim 19 further comprising the step ofcalibrating said first and second two-dimensional images as a functionof the dimensions of the surroundings within the world.